Data Center Cooling

ABSTRACT

A system for cooling air in a data center includes a data center having electronic equipment in operation, a cooling water source, and a plurality of on-floor cooling units. The cooling water source is configured to retain at maximum capacity a total amount of water. Each on-floor cooling unit is configured to cool air heated by a portion of the electronic equipment in the data center using water from the cooling water source. The total amount of water is insufficient to maintain a leaving air temperature of each on-floor cooling unit below an inside setpoint when a temperature outside of the data center is above a predetermined external temperature.

TECHNICAL FIELD

This document relates to systems and methods for providing cooling forareas containing electronic equipment, such as computer server rooms andserver racks in computer data centers.

BACKGROUND

Computer users often focus on the speed of computer microprocessors(e.g., megahertz and gigahertz). Many forget that this speed often comeswith a cost—higher electrical power consumption. For one or two homePCs, this extra power may be negligible when compared to the cost ofrunning the many other electrical appliances in a home. But in datacenter applications, where thousands of microprocessors may be operated,electrical power requirements can be very important.

Power consumption is also, in effect, a double whammy. Not only must adata center operator pay for electricity to operate its many computers,but the operator must also pay to cool the computers. That is because,by simple laws of physics, all the power has to go somewhere, and thatsomewhere is, for the most part, conversion into heat. A pair ofmicroprocessors mounted on a single motherboard can draw 200-400 wattsor more of power. Multiply that figure by several thousand (or tens ofthousands) to account for the many computers in a large data center, andone can readily appreciate the amount of heat that can be generated. Itis much like having a room filled with thousands of burning floodlights.

Thus, the cost of removing all of the heat can also be a major cost ofoperating large data centers. That cost typically involves the use ofeven more energy, in the form of electricity and natural gas, to operatechillers, condensers, pumps, fans, cooling towers, and other relatedcomponents. Heat removal can also be important because, althoughmicroprocessors may not be as sensitive to heat as are people, increasesin heat generally can cause great increases in microprocessor errors andfailures. In sum, such a system may require electricity to run thechips, and more electricity to cool the chips.

SUMMARY

In general, in one aspect, a system for providing cooled air to a datacenter includes a data center having electronic equipment in operation,a cooling water source, a plurality of on-floor cooling units in thedata center, a plurality of proportioning control valves, and acontroller. Each on-floor cooling unit is configured to cool air warmedby a sub-set of the electronic equipment in the data center. Eachproportioning control valve is associated with a single on-floor coolingunit. Each proportioning control valve is configured to turn in responseto a signal from the controller, the valve able to be fully closed,fully open, or partially open. When the valve is closed, water isblocked from the on-floor cooling unit. When the valve is fully open, amaximum volume of water is circulated through the on-floor cooling unit.When the valve is partially open, some percent less than 100% of themaximum amount of water is circulated through the on-floor cooling unit.The controller is configured to turn the corresponding control valve inresponse to a change in temperature.

This and other embodiments can optionally include one or more of thefollowing features. The controller can be configured to turn thecorresponding control valve in response to a change in a leaving airtemperature of the corresponding on-floor cooling unit. The controllercan be configured to turn the corresponding control valve such that theleaving temperature of the corresponding on-floor cooling unit remainsbelow an inside setpoint for at least 90% of the operating time of thesystem. The system can further include a plurality of sensors, eachsensor configured to measure the leaving air temperature of an on-floorcooling unit. The proportioning valve can act independently of any othercontrol valve. The cooling water source can include a cooling tower.Each on-floor cooing unit can include a heat exchanger configured totransfer heat from the electronic equipment to the cooling water source.The heat exchange can include coils located adjacent to one or morecommon hot air plenums that receive heated air from the electronicequipment.

In general, in one aspect, a system for providing cooled air toelectronic equipment includes a data center having electronic equipmentin operation, a plurality of modules connected by a first head, and atleast one chiller connected by a second header. Each module includes aplurality of on-floor cooling units in the data center and a coolingwater source. The at least one chiller is in fluid connection with oneor more than one module in the plurality of modules.

This and other embodiments can optionally include one or more of thefollowing features. The number of chillers can be less than the numberof modules. Each on-floor cooling unit can include a correspondingproportioning control valve configured to control the flow of waterthrough the on-floor cooling unit. The cooling water source can be acooling tower. Each on-floor cooling unit can include a heat exchangerconfigured to transfer heat from the electronic equipment to the coolingwater source. The heat exchanger can include coils located adjacent toone or more common hot air plenums that receive heated air from theelectronic equipment.

In general, in one aspect, a system for cooling air in a data centerincludes a data center having electronic equipment in operation, acooling water source, and a plurality of on-floor cooling units. Thecooling water source is configured to retain at maximum capacity a totalamount of water. Each on-floor cooling unit is configured to cool airheated by a portion of the electronic equipment in the data center usingwater from the cooling water source. The total amount of water isinsufficient to maintain a leaving air temperature of each on-floorcooling unit below an inside setpoint when a temperature outside of thedata center is above a predetermined external temperature.

This and other embodiments can optionally include one or more of thefollowing features. The cooling water source can be a cooling tower.Each on-floor cooling unit can include a heat exchanger configured totransfer heat from the electronic equipment to the cooling water source.The heat exchanger can include coils located adjacent to one or morecommon hot air plenums that receive heated air from the electronicequipment. The system can be is located in a geographical region, andfor at least 90% of a year, temperatures outside of the data center canbe below the predetermined external temperature. At least 95% of theyear, temperatures outside of the data center can be below thepredetermined external temperature.

Advantages of the systems and methods described herein may include oneor more of the following. If the temperature inside the data center isallowed to rise above the inside setpoint for short periods of time,then the use of expensive chillers can be minimized. By monitoring theleaving air temperature, e.g. the temperature of the air leaving a thecooling coil or cool air plenum of the on-floor cooling unit, ratherthan the temperature of individual servers, the temperature indicationwill be more accurate, making the cooling system more efficient. Ifcontrol valves are configured to individually control the flow of waterto each on-floor cooling unit, the total amount of water required forthe system can be reduced. By connecting chillers to a separate header,corresponding modules, e.g. modules which include a series of coolingunits, control valves, and a modular cooling plant, can remain on-lineeven when the corresponding modular cooling plant fails.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features and advantages willbe apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram showing a system for cooling a computerdata center.

FIG. 2A is a schematic diagram showing a system for cooling server racksin a data center.

FIG. 2B is a schematic diagram showing two different plants, each havinga system for cooling server racks in a data center.

FIG. 3 is a psychometric chart showing a heating and cooling cycle forair in a data center.

FIG. 4 is a graph of setpoint temperature for a computing facility overa one year time period.

FIG. 5 is a flowchart showing steps for cooling a data center bymeasuring the leaving air temperature and accounting for elevatedtemperatures during limited times of the year.

FIG. 6 is a flowchart showing steps for cooling a data center using oneor more periods of elevated temperatures for less than 90% of theoperating time of the data center electronic equipment.

FIG. 7 is a flowchart for cooling a data center having both chilledwater and cooled water and using one or more periods of elevatedtemperatures

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a schematic diagram showing a system 100 for cooling acomputer data center 101. The system 100 generally includes an on-floorcooling unit 160 having an air handling unit (including e.g., fan 110and cooling coils 112 a, 112 b) for transferring heat from the datacenter's air to cooling water. System 100 can also include a modularcooling plant 222. Modular cooling plant 222 can include a power andcooling unit (“PCU”) having pumps 124, 120, valves 134, filters (notshown), and a heat exchanger 122 for removing heat from the coolingwater and passing it to condenser water that is delivered to a coolingtower 118 in modular cooling plant 222. The cooling tower 118 in modularcooling plant 222 a cooling water tower, a dry cooler including only afan coil unit, or a hybrid tower including both a cooling water towerand a dry cooler. Alternatively, a cool water source such as a lake orbay can be used instead of a cooling tower 118. The cooling water 118 inmodular cooling plant 222 passes the accumulated heat to the ambient airthrough evaporation and cooling of the cooling tower water to createcooled water. In general operation, the system 100 operates using onlythe cooling tower/heat exchanger/cooling coil system, though a poweredrefrigeration system such as a chiller may be used to provide chilledwater during peak loads, such as when the outdoor ambient dew point isvery high and the cooling tower cannot provide sufficient cooling alone.As explained below, control parameters for the system may also be set soas to avoid most or any need for the use of chillers or other suchpowered refrigeration systems.

The temperatures of each portion of the system 100 are selected to berelatively high (compared to conventional refrigeration-based coolingsystems), so as to permit more efficient operation of the system 100.For example, relatively high air temperatures in the system (e.g., airentering a cooling coil over 110° F. (43.3° C.) and exiting the coil ata temperature above 70° F. (21.11° C.)) may in turn permit forrelatively high cooling water temperatures (e.g., water entering acooling coil around 68° F. (20° C.) and exiting around 104° F. (40° C.))because the amount of heat that can be taken out of the air is generallyproportional to the difference in temperature between the water and theair. If the difference can be kept at an acceptable level, where thetemperatures are high enough that evaporative cooling (e.g., coolingthrough a cooling tower, without further cooling via chiller) issufficient, the relatively high electrical cost of operating a chiller(or many chillers) may be avoided.

High system temperatures may be particularly advantageous in certainimplementations when hybrid cooling towers are used. Such hybrid coolingtowers combine the functionality of an ordinary cooling tower with awater-to-water heat exchanger. Using a sufficiently high temperaturesetpoint, i.e. the maximum temperature at which the data center 101 isallowed to operate for the majority of the operating time of theelectronic equipment, may allow the hybrid tower to provide substantialcooling capacity, even when operating in a water-to-air mode withoututility water. As a result, a hybrid cooling tower may be used toprovide cooling capacity to a facility relatively quickly, even beforeutility water may be obtained in large volumes. The capacity of thecooling tower may be directly related to the difference in thetemperature of the water within it to the ambient outside air.

When the difference in temperatures is not very large, a change of onlya few degrees can bring substantial gains in efficiency. For example,where the cooling water enters at 68° F. (20° C.), by heating air to113° F. (45° C.) rather than 104° F. (40° C.), the temperaturedifference is increased from 36° F. to 45° F. (20° C. to 45° C.)—whichmay result in an increase in heat flow of 25 percent. The actualdifference will vary slightly, as the entering conditions for air andwater are not the only conditions (because the air cools as it passesthrough a cooling coil, and the water warms); this example, however,indicates how the difference in temperature can affect efficiency of asystem.

Use of elevated temperatures in a system may also prevent air in oraround the system from falling below its liquid saturation point, i.e.,its dew point, and condensing. This may, in certain circumstances,provide benefits both in efficiency and in operation of the system.Efficiency benefits may be obtained because creating condensationrequires much more energy than simply cooling air, so that systemscreating condensation may use a large amount of electricity or otherenergy. Improvements in operation of the system may occur because, ifpipes in the system carry water that is below the saturation temperatureof the air around the pipes, condensation might form on the pipes. Thatcondensation can damage the pipes or equipment in the conditioned space,cause mold, and cause water to pool on the floor, and can require theinstallation of insulation on the pipes (to stop the condensation).

In the system shown in FIG. 1, use of elevated temperatures maysubstantially reduce, or almost entirely eliminate, the need forenergy-intensive cooling components such as chillers and the like, evenwhere the heat load in the data center 101 is very high. As a result,system 100 may be operated at a lower operating cost than couldotherwise be achieved. In addition, lower capital costs may be required,because fans, coils, heat exchangers, and cooling towers are relativelybasic and inexpensive components. In addition, by operating with ahigher temperature difference between cooled air and cooling water, lessvolume of cooling water is needed, thus reducing the size and cost ofpiping, and the cost to operate pumps and other such components.

In addition, those components are often very standardized, so that theiracquisition costs are lower, and they are more easily located,particularly in developing countries and remote areas where it may bebeneficial to place a data center 101. Use of system 100 in remote areasand other areas with limited access to electrical power is also helpedby the fact that system 100 may be operated using less electrical power.As a result, such a system can be located near lower-power electricalsub-stations and the like. As discussed more completely below,lower-powered systems may also be amenable to being implemented asself-powered systems using energy sources such as solar, wind,natural-gas powered turbines, fuel cells, and the like.

Referring now to FIG. 1, there is shown a data center 101 in sectionalview, which as shown, is a building that houses a large number ofcomputers or similar heat-generating electronic components. A workspace106 is defined around the computers, which are arranged in a number ofparallel rows and mounted in vertical racks, such as racks 102 a, 102 b.The racks may include pairs of vertical rails to which are attachedpaired mounting brackets (not shown). Trays containing computers, suchas standard circuit boards in the form of motherboards, may be placed onthe mounting brackets.

In one example, the mounting brackets may be angled rails welded orotherwise adhered to vertical rails in the frame of a rack, and traysmay include motherboards that are slid into place on top of thebrackets, similar to the manner in which food trays are slid ontostorage racks in a cafeteria, or bread trays are slid into bread racks.The trays may be spaced closely together to maximize the number of traysin a data center, but sufficiently far apart to contain all thecomponents on the trays and to permit air circulation between the trays.

Other arrangements may also be used. For example, trays may be mountedvertically in groups, such as in the form of computer blades. The traysmay simply rest in a rack and be electrically connected after they areslid into place, or they may be provided with mechanisms, such aselectrical traces along one edge, that create electrical and dataconnections when they are slid into place.

Air may circulate from workspace 106 across the trays and throughon-floor cooling unit 160. Although only one on-floor cooling unit 160is shown in FIG. 160, data center 101 can include multiple on-floorcooling units 160. On-floor cooling unit 160 includes warm-air plenums104 a, 104 b behind the trays. The air may be drawn into the trays byfans mounted at the back of the trays (not shown). The fans may beprogrammed or otherwise configured to maintain a set exhaust temperaturefor the air into the warm air plenum, and may also be programmed orotherwise configured to maintain a particular temperature rise acrossthe trays. Where the temperature of the air in the work space 106 isknown, controlling the exhaust temperature also indirectly controls thetemperature rise. The work space 106 may, in certain circumstances, bereferenced as a “cold aisle,” and the plenums 104 a, 104 b as “warmaisles.”

The temperature rise can be large. For example, the work space 106temperature may be about 77° F. (25° C.) and the exhaust temperatureinto the warm-air plenums 104 a, 104 b may be set to 113° F. (45° C.),for a 36° F. (20° C.)) rise in temperature. The exhaust temperature mayalso be as much as 212° F. (100° C.) where the heat generating equipmentcan operate at such elevated temperature. For example, the temperatureof the air exiting the equipment and entering the warm-air plenum may be118.4, 122, 129.2, 136.4, 143.6, 150.8, 158, 165, 172.4, 179.6, 186.8,194, 201, or 208.4° F. (48, 50, 54, 58, 62, 66, 70, 74, 78, 82, 86, 90,94, or 98° C.). Such a high exhaust temperature generally runs contraryto teachings that cooling of heat-generating electronic equipment isbest conducted by washing the equipment with large amounts offast-moving, cool air. Such a cool-air approach does cool the equipment,but also uses a lot of energy.

Cooling of particular electronic equipment, such as microprocessors, maybe improved even where the flow of air across the trays is slow, byattaching impingement fans to the tops of the microprocessors or otherparticularly warm components, or by providing heat pipes and relatedheat exchangers for such components.

The heated air may be routed upward into a ceiling area, or attic 105,or into a raised floor or basement, or other appropriate space, and maybe gathered there by air handling units that include, for example, fan110 of on-floor cooling unit 160, which may include, for example, one ormore centrifugal fans appropriately sized for the task. The fan 110 maythen deliver the air back into a plenum 108 located adjacent to theworkspace 106. The plenum 108 may be simply a bay-sized area in themiddle of a row of racks, that has been left empty of racks, and thathas been isolated from any warm-air plenums on either side of it, andfrom cold-air work space 106 on its other sides. Alternatively, air maybe cooled by coils defining a border of warm-air plenums 104 a, 104 band expelled directly into workspace 106, such as at the tops ofwarm-air plenums 104 a, 104 b.

On-floor cooling unit 160 can also have cooling coils 112 a, 112 blocated on opposed sides of the plenum approximately flush with thefronts of the racks. (The racks in the same row as the plenum 108,coming in and out of the page in the figure, are not shown.) The coilsmay have a large surface area and be very thin so as to present a lowpressure drop to the system 100. In this way, slower, smaller, andquieter fans may be used to drive air through the system. Protectivestructures such as louvers or wire mesh may be placed in front of thecoils 112 a, 112 b to prevent them from being damaged.

In operation, fan 110 of on-floor cooling unit 160 pushes air down intoplenum 108, causing increased pressure in plenum 108 to push air outthrough cooling coils 112 a, 112 b. As the air passes through the coils112 a, 112 b, its heat is transferred into the water in the coils 112 a,112 b, and the air is cooled.

The speed of the fan 110 and/or the flow rate or temperature of coolingwater flowing in the cooling coils 112 a, 112 b may be controlled inresponse to measured values. For example, the pumps driving the coolingliquid may be variable speed pumps that are controlled to maintain aparticular temperature in work space 106. Such control mechanisms may beused to maintain a constant temperature in workspace 106 or plenums 104a, 104 b and attic 105.

The workspace 106 air may then be drawn into racks 102 a, 102 b such asby fans mounted on the many trays that are mounted in racks 102 a, 102b. This air may be heated as it passes over the trays and through powersupplies running the computers on the trays, and may then enter thewarm-air plenums 104 a, 104 b. Each tray may have its own power supplyand fan or fans. In some implementation, the power supply is located atthe back edge of the tray, and the is fan attached to the back of thepower supply. All of the fans may be configured or programmed to deliverair at a single common temperature, such as at a set 113° F. (45° C.).The process may then be continuously readjusted as fan 110 captures andcirculates the warm air.

Additional items may also be cooled using system 100. For example, room116 is provided with a self-contained fan-coil unit 114 which contains afan and a cooling coil. The unit 114 may operate, for example, inresponse to a thermostat provided in room 116. Room 116 may be, forexample, an office or other workspace ancillary to the main portions ofthe data center 101.

In addition, supplemental cooling may also be provided to room 116 ifnecessary. For example, a standard roof-top or similar air-conditioningunit (not shown) may be installed to provide particular cooling needs ona spot basis. As one example, system 100 may be designed to deliver 78°F. (25.56° C.) supply air to work space 106, and workers may prefer tohave an office in room 116 that is cooler. Thus, a dedicatedair-conditioning unit may be provided for the office. This unit may beoperated relatively efficiently, however, where its coverage is limitedto a relatively small area of a building or a relatively small part ofthe heat load from a building. Also, cooling units, such as chillers,may provide for supplemental cooling, though their size may be reducedsubstantially compared to if they were used to provide substantialcooling for the system 100.

Fresh air may be provided to the workspace 106 by various mechanisms.For example, a supplemental air-conditioning unit (not shown), such as astandard roof-top unit may be provided to supply necessary exchanges ofoutside air. Also, such a unit may serve to dehumidify the workspace 106for the limited latent loads in the system 100, such as humanperspiration. Alternatively, louvers may be provided from the outsideenvironment to the system 100, such as powered louvers to connect to thewarm air plenum 104 b. System 100 may be controlled to draw air throughthe plenums when environmental (outside) ambient humidity andtemperature are sufficiently low to permit cooling with outside air.Such louvers may also be ducted to fan 110, and warm air in plenums 104a, 104 b may simply be exhausted to atmosphere, so that the outside airdoes not mix with, and get diluted by, the warm air from the computers.Appropriate filtration may also be provided in the system, particularlywhere outside air is used.

Also, the workspace 106 may include heat loads other than the trays,such as from people in the space and lighting. Where the volume of airpassing through the various racks is very high and picks up a very largethermal load from multiple computers, the small additional load fromother sources may be negligible, apart from perhaps a small latent heatload caused by workers, which may be removed by a smaller auxiliary airconditioning unit as described above.

Cooling water may be provided from modular cooling plant 222 that caninclude a cooling water circuit and a condenser water circuit. Coolingwater circuit can be powered by pump 124. The cooling water circuit maybe formed as a direct-return, or indirect-return, circuit, and maygenerally be a closed-loop system. Pump 124 may take any appropriateform, such as a standard centrifugal pump. Heat exchanger 122 may removeheat from the cooling water in the circuit. Heat exchanger 122 may takeany appropriate form, such as a plate-and-frame heat exchanger or ashell-and-tube heat exchanger.

Heat may be passed from the cooling water circuit to a condenser watercircuit that includes heat exchanger 122, pump 120, and cooling tower118. Pump 120 may also take any appropriate form, such as a centrifugalpump. Cooling tower 118 may be, for example, one or more forced drafttowers or induced draft towers. The cooling tower 118 may be considereda free cooling source, because it requires power only for movement ofthe water in the system and in some implementations the powering of afan to cause evaporation; it does not require operation of a compressorin a chiller or similar structure.

The cooling tower 118 may take a variety of forms, including as a hybridcooling tower. Such a tower may combine both the evaporative coolingstructures of a cooling tower with a water-to-water heat exchanger. As aresult, such a tower may be fit in a smaller face and be operated moremodularly than a standard cooling tower with separate heat exchanger.Additional advantage may be that hybrid towers may be run dry, asdiscussed above. In addition, hybrid towers may also better avoid thecreation of water plumes that may be viewed negatively by neighbors of afacility.

As shown, the fluid circuits may create an indirect water-sideeconomizer arrangement. This arrangement may be relatively energyefficient, in that the only energy needed to power it is the energy foroperating several pumps and fans. In addition, this system may berelatively inexpensive to implement, because pumps, fans, coolingtowers, and heat exchangers are relatively technologically simplestructures that are widely available in many forms. In addition, becausethe structures are relatively simple, repairs and maintenance may beless expensive and easier to complete. Such repairs may be possiblewithout the need for technicians with highly specialized knowledge.

Alternatively, direct free cooling may be employed, such as byeliminating heat exchanger 122, and routing cooling tower water(condenser water) directly to cooling coils 112 a, 112 b (not shown).Such an implementation may be more efficient, as it removes one heatexchanging step. However, such an implementation also causes water fromthe cooling tower 118 to be introduced into what would otherwise be aclosed system. As a result, the system in such an implementation may befilled with water that may contain bacteria, algae, and atmosphericcontaminants, and may also be filled with other contaminants in thewater. A hybrid tower, as discussed above, may provide similar benefitswithout the same detriments.

Control valve 126 is provided in the condenser water circuit to supplymake-up water to the circuit. Make-up water may generally be neededbecause cooling tower 118 operates by evaporating large amounts of waterfrom the circuit. The control valve 126 may be tied to a water levelsensor in cooling tower 118, or to a basin shared by multiple coolingtowers. When the water falls below a predetermined level, control valve126 may be caused to open and supply additional makeup water to thecircuit. A back-flow preventer (BFP) may also be provided in the make-upwater line to prevent flow of water back from cooling tower 118 to amain water system, which may cause contamination of such a water system.

Although FIG. 1 shows the modular cooling plant 222 connected to onlyone on-floor cooling unit 160, a modular cooling plant 222 can beconnected to a number of on-floor cooling units 160, as shown in FIG.2A, with the on-floor cooling units 160 connected in parallel to themodular cooling plant 222. Each modular cooling plant 222 can serve, forexample, 12 or more on-floor cooling units 160. Furthermore, system 100can have a number of modular cooling plants 222. System 100 can be, forexample, a building using 30 megawatts (MW) of electrical power. Therecan then be a number of modular cooling plants 222 connected to thebuilding, for example fifteen modular cooling plants 222 each providingcooling for each 2 MW of electrical power used by the data center.

The modular cooling plants 222 and on-floor cooling units 222 can beconnected through ladder pipes 244. Supply line 246 can supply coolwater to the on-floor cooling units, while return line 248 can returnwarm water to the modular cooling plants 222 to be cooled. Along atleast one pipe is a corresponding control valve 224 located so that flowto a single cooling unit can be controlled. The modular cooling plant222, corresponding cooling units 160, and control valves 224 cantogether be called a module 230. Each control valve 224 can control theflow of water through the corresponding on-floor cooling unit 160without affecting the other cooling units in the module. The controlvalves 224 can provide only the water that the particular on-floorcooling unit 160 needs. That is, the control valve 224 can be aproportioning valve rather than a digital “on” or “off” type controlvalve. Each control valve 224 can have a motor (not shown) that can turnthe corresponding valve such that it is open as wide as necessary tocirculate the needed water through the corresponding on-floor coolingunit 160. A controller can control the motor. There can be, for example,one master controller to control all of the control valves 224.Alternatively, each control valve 224 can have a correspondingcontroller to independently control each motor. In some implementations,a master controller can control all of the individual controllers. Thecontrollers can control the control valves 224 to respond to the leavingair temperature (LAT) or other local variables of each cooling unit 160.For example, the controllers can control the control valves 224 torespond to a change in the difference between the leaving airtemperature and the entering water temperature. Each control valve 224,and thus each on-floor cooling unit 160, can act independently ofanother valve 224 or on-floor cooling unit 160. That is, all increasesor decreases in flow through system 100 can be local through controllersat each corresponding on-floor cooling unit 160.

By designing system 100 such that the amount of water to each on-floorcooling unit 160 is individually controlled and such that all of themodular cooling plants 222 are connected to a common header, water fromthe system can be distributed to the load in system 100 rather thanflowing a constant amount of water through the entire system 100. Inthis way, water that is not needed for one on-floor cooling unit 160 canbe made available to another on-floor cooling unit 160. Because theon-floor cooling units 160 act independently, there can be a largenumber of on-floor cooling units 160 in system 100, for example over1,000 cooling units. Further, the system 100 can be designed such thatcooled air from multiple cooling units 160 can mix between the time itleaves cooling coils 112 a, 112 b and is drawn into the servers 102 a,102 b, such as by using higher ceiling to exhaust the cooled air fromthe fan 110 and cooling coils 112 a,112 b into the space above theservers 102 a, 102 b. Having such a design ensures that if one on-floorcooling unit 160 fails, e.g. as a result of control valve 224 flowingtoo much or too little water to the respective on-floor cooling unit160, the other on-floor cooling units 160 can compensate to cool thedata center 101. Moreover, system 100 can be designed such that if onemodular cooling plant 222 fails, other modular cooling plants 222 alongthe same header can take the load. Such a design can allow for thecooling of large numbers of servers. For example, system 100 can includeover 1,000 cooling units and corresponding server racks.

Optionally, a few chillers 130 can be available for use by the system100. A separate chiller header 228 may be provided which connects onechiller to multiple modular cooling plants 222. Operation of system 100may switch some or all of the chillers 130 on during times of extremeatmospheric ambient (i.e., hot and humid) conditions or times of highheat load in the data center 101. Referring back to FIG. 1, controlledmixing valves 134 are provided for electronically switching to thechiller circuit, or for blending cooling water from the chiller circuitwith cooling water from the condenser circuit. Pump 128 is able tosupply tower water to chiller 130, and pump 132 is able to supplychilled water, or cooling water, from chiller 130 to the remainder ofsystem 100. Chiller 130 can take any appropriate form, such as acentrifugal, reciprocating, or screw chiller, or an absorption chiller.

The chiller circuit can be controlled to provide various appropriatetemperatures for cooling water. The chilled water may be supplied fromchiller 130 at temperatures elevated from conventional chilled watertemperatures. For example, the chilled water may be supplied attemperatures of 55° F. (13° C.) to 70° F. (18 to 21° C.) or higher. Forexample, the water supply temperature can be between 60-64° F., 65-70°F., 71-75° F., or 76-80° F. The water may then be returned at highertemperatures, such as 59 to 176° F. (15 to 80° C.). In this approachthat uses sources in addition to, or as an alternative to, free cooling,increases in the supply temperature of the chilled water can also resultin substantial efficiency improvements for the system 100.

Referring to FIG. 2A, because the chillers 130 are on a separate circuit228 from the modules 230, the chillers 130 can be shared among modules230. Such an arrangement can be beneficial when a modular cooling plant222 or chiller 130 fails. Because the chillers 130 are shared amongmodules 160, any chiller 130 can supply chilled water to keep the module160 functioning when either one chillers fails or when a correspondingmodular cooling plant 222 fails. Thus, the system 100 can become morerobust as more modules 160 and more chillers 130 are added, as describedfurther herein. Furthermore, the number of chillers 130 can be less thanthe number of modules 230 to reduce costs. Valves between the chillersand the modular cooling plants 222 can control which chiller supplieschilled water to which modular cooling plant 222. Although not shown,and although on separate headers from the modules 230, the chillers 130can be housed in some or all of the modules 222.

Referring back to FIG. 1, pumps 120, 124, 128, 132, may be provided withvariable speed drives. Such drives may be electronically controlled by acentral control system to change the amount of water pumped by each pumpin response to changing environmental conditions or changing conditionssuch as equipment failure or a new set point in system 100. For example,pump 124 may be controlled to maintain a particular temperature inworkspace 106, such as in response to signals from a thermostat or othersensor in workspace 106.

As shown in FIG. 2B, system 100 can be highly modular, and can thereforebe scaled from a very small system to a very large system by addingadditional components and subsystems. Because system 100 is modular, thedata center can be expanded as capacity is needed while maintainingoperation of the plant during construction. A first set of racks ofelectronic equipment, for example, racks of servers, and associatedmodular cooling plants and optional chillers can be installed andoperation of the equipment can be initiated. As more electronicequipment is needed and received, a second set of the electronicequipment and associated modular cooling plants are installed and placedonline. The modular cooling plants associated with the first set ofequipment can be fluidly coupled to the modular cooling plantsassociated with the second set of equipment to provide the backupcooling in the event of failover. Furthermore, although existing plant301 and future plant 302 are shown as separate systems, connections,such as connection 303, can be made between the two plants in order toshare components. While sharing components between old and new systemsmay cause construction-based interruptions in the existing system, itmay also provide for better utilization of the components in the fullsystem. For example, chillers 130 can be extended to be shared amongboth plants 301 and 302.

In operation, system 100 may respond to signals from one or more sensors192 placed in system 100. The sensors may include, for example,thermostats, humidistats, flowmeters, and other similar sensors. In oneimplementation, one or more thermostats may be provided to monitor thetemperature inside the data center. In some embodiments, air adjacent toan on-floor cooling unit 160 is partially isolated from air adjacent toa neighboring on-floor cooling unit 160. A single on-floor cooling unit160 cools heated air generated by the servers in the associatedworkspace. The sensors 192 can measure the leaving air temperature,defined as the temperature leaving the cool air coils 112 a, 112 b orthe cool air plenum 108. To measure the leaving air temperature, the oneor more sensors 192 can be placed, for example, next to he cool aircoils 112 a, 112 b or the cool air plenum 108 of each on-floor coolingunit 160.

As shown in FIG. 1, a single temperature sensor 192 can be used for eachon-floor cooling unit 160. Alternatively, an array of sensors, such asbetween two and ten sensors, for example four sensors, may be used foreach on-floor cooling unit and the average temperature, such as theaverage leaving air temperature, determined. While other implementationsmay include placing multiple thermostats throughout system 110, such asin the warm air plenums 104 a, 104 b or near the servers 102 a, suchtemperatures tend to vary. Measuring the leaving air temperature maytherefore be a more robust method of controlling system 100.

The leaving air temperature measured by the thermostats can be used tocontrol the valves 224 associated with the on-floor cooling units 160.Further, although in some implementations, the pumps 120, 124, 128, 132are fixed speed pumps, in other implementations, the temperature readingfrom the thermostats may be used to control the speed of associatedpumps. When the leaving air temperature begins to rise above an insidesetpoint, the control valves 224 can be open wider or the pumps 120,124, 128, 132 can run faster to provide additional cooling water. Suchadditional cooling water can reduce the leaving air temperature. Theleaving air temperature can also be used to indirectly effect the serverfans. That is, for computers that regulate their fan speed based on howhard the server is working, controlling the cooling unit leave airtemperature to eliminate hot spots in the work area 106 can allow thoseservers to run their fans slower, thereby using less energy that wouldotherwise be required.

When additional water is circulated through system 100 as a result ofthe valves 224 opening or the pumps 120, 124, 128, 132 pulling morewater, the pressure differential between the supply 246 and return 248ladders falls, requiring the pumps 120, 124, 128, 132 to settle on alower pressure equilibrium than usual (called “riding the pump curve”),whereby the pressure drop across the cooling units matches the pressuredifferential of the pumps. As excess water is drawn from modular coolingplant 222, the modular cooling plant 222 may not be able to maintain apredetermined water supply temperature. For example, because the wateris cycling through the system more quickly and the environmentaltemperature is higher, the cooling towers may not have sufficient timeto kick off heat from the water to the environment than when theexternal temperature is lower. As a result, water that is warmer thanthe predetermined water supply temperature is circulated through system100.

The potentially negative result of circulating water that is warmer thanthe predetermined water supply temperature through an on-floor coolingunit 160, that is, of less cooling ability of the water flowing throughthe cooling unit 160, however, can be more than offset by the positiveresult of flowing extra water through the on-floor cooling unit 160.Thus, the resulting leaving air temperature can be brought down belowthe inside setpoint. In some embodiments, while the cooling watersupplied to on-floor cooling units 160 may be 1 degree warmer than thepredetermined water supply temperature, the resulting leaving airtemperature may be more than 1 degree cooler than the inside setpoint.

Controllers may also be used to control the speed of various items suchas fan 110 to maintain a set pressure differential between two spaces,such as attic 105 and workspace 106, and to thereby maintain a desiredairflow rate. Where mechanisms for increasing cooling, such as speedingthe operation of pumps, are no longer capable of keeping up withincreasing loads, a control system may activate chiller 130 andassociated pumps 128, 132, and may modulate control valves 134accordingly to provide additional cooling.

System 100 may also respond to signals from outside data center 101,e.g. to the temperature outside the data center 101 (which cancorrelated to and monitored by the temperature of water supplied fromthe cooling tower). Thermostats may be used to monitor the temperatureoutside the data center 101. When the temperature, e.g. either the airtemperature or the wetbulb temperature, outside of the data center 101is below a predetermined value, water that is cooled in the coolingtower is at least as cool as a predetermined water supply temperature.This cooled water can be circulated through system 100 in order to keepthe temperature inside the data center 101, for example the leaving airtemperature of each on-floor cooling unit 160, below an inside setpoint.The inside setpoint can be, for example, less than 120° F., such as lessthan 115° F. or less than 110° F., such as 75° F. to 85° F. and can varythroughout the year, as discussed below. Federal OSHA and CaliforniaOSHA guidelines may also provide limitations on the inside setpoint.Furthermore, the setpoint might vary by geographical location to takeinto account the average temperature and humidity in the respectivelocation. When the temperature outside the data center 101 rises abovethe predetermined value, water that is warmer than the predeterminedwater supply temperature can be circulated through system 100, thuscausing the temperature inside the data center 101 to rise above theinside setpoint. The data center 101 can be rated to run on water havinga temperature that is less than the predetermined water supplytemperature, i.e. data center 101 may have sufficient water resources tokeep the leaving air temperature below a predetermined temperature andin some cases, to keep the water temperature in system 100 less than thepredetermined water supply temperature.

In one implementation, supply temperatures for cooling water can bebetween 65° F. and 70° F., such as 68° F. (20° C.), while returntemperatures can be between 100° F. and 110° F., such as 104° F. (40°C.). In other implementations, the supply water can be supplied attemperatures of 50° F. to 84.20° F. or 104° F. (10° C. to 29° C. or 40°C.) and the return water can be supplied at temperatures of 59° F. to176° F. (15° C. to 80° C.) for return water. The temperature of thewater supplied by the cooling tower may be generally slightly above thewet bulb temperature under ambient atmospheric conditions, while thetemperature of the water returned to the cooling tower will depend inpart on the heat inside building 101.

Using these parameters and the parameters discussed above for enteringand exiting air, relatively narrow approach temperatures may be achievedwith the system 100. The approach temperature, in this example, is thedifference in temperature between the air flowing away from a coil andthe water entering a coil. The approach temperature is always positivebecause the water entering the coil is the coldest water, and the waterwarms as it travels through the coil. As a result, the water can beappreciably warmer by the time it exits the coil. Air passing over thecoil near the water's exit point is warmer than air passing over thecoil at the water's entrance point. Because even the most-cooled exitingair at the cooling water's entrance point is warmer than the enteringwater, the overall exiting air temperature is at least somewhat warmerthan the entering cooling water temperature.

Keeping the approach temperature small permits a system to be run onfree, or evaporative, cooling for a larger portion of the year andreduces the size of a chiller used with the system, if any chiller isneeded at all. To lower the approach temperature, the cooling coils maybe designed for counterflow. In counter-flow, the warmest air flows nearthe warmest water and the coolest air exits near where the coolest waterenters.

In certain implementations, the entering water temperature may be 64° F.(18° C.) and the exiting air temperature 77° F. (25° C.), as notedabove, for an approach temperature of 13° F. (7° C.). In otherimplementations, wider or narrower approach temperature may be selectedbased on economic considerations for an overall facility.

With a close approach temperature, the temperature of the cooled airleaving the coil closely tracks the temperature of the cooling waterentering the coil. As a result, the air temperature can be maintained,generally regardless of load, by maintaining a constant watertemperature. In an evaporative cooling mode, a constant watertemperature can be maintained as the wet bulb temperature stays constant(or changes very slowly), and by blending warmer return water withsupply water or modulating cooling tower fans as the wet bulbtemperature falls. As such, active control of the cooling airtemperature can be avoided in certain situations, and control can occursimply on the cooling water return and supply temperatures.

FIG. 3 is a psychrometric chart showing a heating and cooling cycle forair in a data center. A psychrometric chart graphically represents thethermodynamic properties of moist air (which is air containing anyappreciable moisture, and not merely air that would feel moist to aperson). The chart is from ASHRAE Psychrometric Chart No. 1, whichdefines properties of air for sea level applications. See 1997 ASHRAEHandbook—Fundamentals, at page 6.15. Other charts may also be used, andthe chart shown here is merely used to exemplify certain aspects of theconcepts discussed in the present document.

The psychrometric chart is criss-crossed with a number of lines thatrepresent various properties of air. Cooling and heating processes onair can be analyzed by identifying a point on the chart that representsair at a particular condition (e.g., temperature and humidity), and thenlocating another point that represents the air at another condition. Aline between those points, generally drawn as a straight line, may befairly assumed to represent the conditions of the air as it moves fromthe first condition to the second, such as by a cooling process.

Several properties will be discussed here. First, saturation temperatureis an arc along the left of the graph and represents the temperature atwhich air becomes saturated and moisture begins coming out of the air asa liquid—also known popularly as the “dew point.” When the temperatureof air is taken below the dew point, more and more water comes out ofthe air because the cooler air is capable of holding less water.

The dry bulb temperature of air is listed along the bottom of the graphand represents what is popularly viewed as temperature, i.e., thetemperature returned by a typical mercury thermometer.

The chart shows two numbers relating to the humidity of air. The firstis the humidity ratio, listed along the right edge of the graph, and issimply the weight of moisture per each unit weight of dry air. Thus, thehumidity ratio will stay constant at various temperatures of air, untilmoisture is removed from the air, such as by pushing the air temperaturedown to its dew point (e.g., the moisture comes out of the air and endsup on the grass in the morning) or by putting moisture into the air(e.g., by atomizing water into such as fine mist in a humidifier thatthe mist can be supported by the natural motion of the air molecules).Thus, when graphing processes that involve simple changes in airtemperature, the point that represents the state of the air will movestraight left and right along the graph at a constant humidity ratio.That is because the dry-bulb temperature will go up and down, but thehumidity ratio will stay constant.

The second humidity parameter is the so-called relative humidity. Unlikethe humidity ratio, which measures the absolute amount of moisture inthe air, the relative humidity measures the amount of moisture in theair as a percent of the total moisture the air could possibly hold atits current temperature. Warmer air can hold more moisture than cancolder air, because the molecules in warmer air are moving more rapidly.Thus, for an equal amount of moisture in the air (i.e., an equalhumidity ratio), the relative humidity will be lower at a hightemperature than at a low temperature.

As one example, on a summer day when the overnight low was 55 degreesFahrenheit and there is dew on the ground, and the daytime high isaround 75 degrees Fahrenheit, the relative humidity in the early morningwill be about 100% (the dew point), but the relative humidity in theafternoon will be a very comfortable 50%, even if one assumes that theidentical amount of water is in the air at both times. This exemplaryprocess is shown in FIG. 3 by the points marked C and D, with point Cshowing saturated air at 55 degrees Fahrenheit (the overnight low), andpoint D representing that same air warmed to 75 degrees Fahrenheit (thedaytime temperature).

Commercial air handling systems take advantage of this same process inproviding conditioned air in a building. Specifically, systems maygather air from an office space at 75 degrees Fahrenheit and a relativehumidity of 60 percent. The systems pass the air through a cooling coilthat looks like an automobile radiator to cool the air to 55 degreesFahrenheit, which will typically push the air down to its dew point.This will make moisture pour out of the air as it passes through thecooling coil. The moisture can be captured in drains below the coolingcoil and then be removed from the building. The air can then be returnedto the work space, and when it warms back to 75 degrees, it will be avery comfortable 50 percent relative humidity.

This common cooling process is shown by points A, B, C, and D on thechart of FIG. 3. Point A shows the 75 degree air at 70 percent relativehumidity. Point B shows the air cooled to its dew point, which it hitsat a temperature (dry bulb) of about 65 degree Fahrenheit. Furthercooling of the air to 55 degrees Fahrenheit (to point C) rides along thesaturation curve, and water will come out of the air during that portionof the cooling. Finally, the state of the air moves to point D as theair is warmed and reaches 75 degrees Fahrenheit again. At this point,the relative humidity will be 50 percent (assuming it does not pick upadditional moisture from the room or the existing room air) rather thanthe original 60 percent because the cooling process has dehumidified theair by pulling moisture out of the air in the cooling coil. If the roomair contains more moisture than does the cooled air, point D will beslightly above its position shown in FIG. 3, but still below point A.

Such a common process brings with it a number of challenges. First, tocool the air to 55 degrees Fahrenheit, the system must provide coolingwater in the cooling coil that can absorb all the heat. Such water wouldneed to be at least cooler than 55 degrees Fahrenheit. It may beexpensive to create such cool water—requiring systems such as chillersand other energy-intensive systems. In addition, the area immediatelyaround the pipes that supply the cooling water will be cooler than 55degrees Fahrenheit, i.e., cooler than the dew point of the air if thepipes run through the workspace or through air having the same state asthe air in the workspace. As a result, moisture from the air maycondense on the pipe because the temperature of the surrounding air hasfallen to its dew point. Thus, insulation may be required around thecool pipes to prevent such condensation, and condensation might occur inany event, and cause rusting, mold, water pooling, or other problems.Finally, it takes a lot of energy to dehumidify, i.e., to change thewater from one state to another.

The warm air cooling features discussed with respect to FIG. 1 abovemay, in certain implementations, avoid one or more of these challenges.An exemplary warm air cooling process is shown on the graph of FIG. 3 bypoints E and F. Point E shows a room-air condition in a workspace thatis near the top of, but within, common guidelines for comfort levels forpeople dressed in summer clothing. See 1997 ASHRAEHandbook—Fundamentals, at page 8.12. That condition is 75° F. (24° C.),and a relative humidity of about 70 percent (the same as Point A in theprior example). Point F shows heating of that air without the additionof moisture, such as by passing the air over heat-generating computercomponents in a rack-mounted server system. The temperature rise is 36°F. (20° C.) to bring the air to a state of 111° F. (44° C.) at about 23percent relative humidity. The air may then be cooled to its originaltemperature (point E) of 75 degrees Fahrenheit in a cooling coil beforeit is re-introduced to the work space, without adding water to orremoving water from the air.

Points G and H on the graph represent the condition of air in the spaceimmediately surrounding cooling pipes. It is assumed for this examplethat the cooling supply water is 68 degrees Fahrenheit (20 degreesCelsius) and the return temperature is 104 degrees Fahrenheit (40degrees Celsius). It is also assumed that the air near the pipes willcontain the same moisture level as the rest of the air in the space, andthat the air immediately surrounding the pipe takes on the sametemperature as the water inside the pipe. As can be seen, this airassociate with the cooling water also stays above the saturation point,so that there should be no condensation on the cooling water pipes, andthus no need for insulation to prevent condensation on the pipes.

It can be seen by this process that the air never becomes saturated. Asa result, the system need not provide energy to create a phase change inthe air. In addition, the system need not provide liquid recoverystructures at the cooling coil, or pipe insulation for anywhere. Othersimilar temperatures, and in many implementations warmer temperatures,may be used. The particular temperatures discussed here are meant to beexemplary only.

FIG. 4 is a graph of an inside setpoint temperature for a computingfacility over a one year time period. The inside setpoint temperaturemay be a temperature in a work space such as work space 106 in FIG. 1.As shown in the stepped graph, the inside setpoint temperature (i.e., atargeted temperature) is adjusted infrequently, such as seasonally ormonthly, so as to more closely track expected outdoor wetbulbtemperatures. The inside setpoint is increased in the summer because thelower winter inside setpoint may be effectively unattainable in warmsummer weather by using only evaporative cooling. Thus, while a “best,”i.e. lower temperature such as 71° F. (22° C.), inside setpointtemperature applies in the winter, that same inside setpoint is notrealistic, in the example, in summer months.

The inside setpoint may be manually set by a user and can be adjustedinfrequently so as to better approximate a setpoint that is attainableusing evaporative cooling techniques with little or no assistance fromchillers or other similar components that require relatively high levelsof energy to operate. Although increasing the inside setpoint duringwarmer times of the year may increase the typical operating temperature,it also decreases the amount of thermal cycling that may occur in afacility, and thus lengthen the life of electronic components in thefacility. In contrast, if the setpoint is kept as low as possible, theconditioned space would be relatively cool on days having a low wet bulbtemperature and relatively warm on days having a high wet bulbtemperature. Thus, keeping a constant setpoint throughout the year mayactually increase thermal cycling, particularly in warmer months—as thesystem is able to maintain the setpoint on some days but not on otherdays. In another embodiment, the cooling units may be controlled tostabilize the difference between their leaving air temperature and theirentering water temperature. By applying resets to the cooling plansetpoints, the air temperature in the work space 106 can be effectivelyslaved to the temperature of the water from the cooling plant, therebyremoving one layer of setpoints to be scheduled.

The setpoint may also be adjusted substantially continuously, such as byvarying the setpoint temperature in an annual sinusoidal manner thatgenerally follows the expected outdoor wet bulb temperature, as shown bythe sinusoidal setpoint line. Studies have indicated that humandiscomfort is minimized by providing many minute changes, or continuouschanges, to temperature, as opposed to large step variations intemperature. In both examples the setpoint may be maintained, in certainimplementations, even if a lower temperature may readily be achieved(e.g., because the outdoor wet bulb temperature is lower than expected)so as to minimize thermal cycling in a facility being cooled.

The particular setpoint temperatures may be selected based on thecapabilities of the components in a facility and on prevailing localweather conditions. For example, cool weather setpoints may be in therange of 59-77° F. (15-25° C.), with particular values of 64.4, 68,71.6, and 75.2° F. (18, 20, 22, and 24° C.). Warm weather setpoints maybe in the range of 68-86° F. (20-30° C.), with particular values of71.6, 75.2, 78.8, and 82.4° F. (22, 24, 26, ad 28° C.). In a particularimplementation, warm weather air temperatures in a facility may beapproximately 80.6-82.4° F. (27-28° C.) and cold weather temperature maybe about 71.60° F. (22° C.). The time for resetting the setpoint mayalso vary, and may be weekly (e.g., using a long range weather forecastto select an achievable setpoint that tracks the predicted wet bulbtemperature), weekly, monthly, or quarterly, for example.

When the wet bulb or air temperature gets too high to achieve thedesired inside set point, the temperature of the cooling water may beallowed to drift upward with the outside temperature, causing thetemperature in space 106 to move upward also. If the outside temperaturerises above a predetermined value, as discussed above, then thetemperature inside the data center 101 can be allowed to rise above aninside setpoint, as shown by area 401 in FIG. 4. This rise in thetemperature inside the data center can be achieved, as discussed above,by flowing warm water, that is water that is warmer than a predeterminedwater supply temperature through system 100. Correspondingly, more watercan be circulated through system 100 in order to carry the heat awayfaster and to compensate for the warm water circulated through thesystem.

The rise in temperature inside the data center 101 over the insidesetpoint can be limited in time, for example to less than 10% of theoperating time of the data center or the equipment, such as less than5%, less than 1%, or less than 0.5% of the operating time. Likewise, theamount of time that warm water is circulated through the system 100 canbe limited in time, for example to less than 1000 hours per year, lessthan 500 hours per year, less than 100 hours per year, or less than 50hours per year. If the temperature inside the data center 101 is overthe inside setpoint for more than 10% of the operating time of the datacenter or the equipment or if warm water is circulated for more than1000 hours per year, the electronic equipment may fail more quickly thandesired. For example, the electronic equipment may fail more quicklyafter running warm water 15% of the annual operating time of the datacenter or equipment or for 1500 hours.

FIG. 5 is a flowchart 500 showing steps for cooling a data center usingone or more periods of elevated temperatures. The method is exemplaryonly; other steps may be added, steps may be removed, and the steps maybe performed in different orders than those shown, as appropriate. Thetemperature outside of the data center is monitored (step 502). Theleaving air temperature from the data center is monitored (step 503). Adetermination is made as to whether the temperature outside the datacenter is above a predetermined value (step 504). When the temperatureoutside of the data center is not above a predetermined value, theleaving air temperature can be maintained below an inside setpoint byflowing cool water through a cooling system within the data center (step505). That is, water that is at least as cool as a predetermined watersupply temperature is circulated through the data center. When thetemperature outside of the data center rises above the predeterminedvalue, the leaving air temperature can be allowed to be greater than theinside setpoint by flowing water through the cooling system that iswarmer than the predetermined water supply temperature (step 506).

FIG. 6 is a flowchart 600 showing steps for cooling a data center usingone or more periods of elevated temperatures for less than 90% of theoperating time of the data center electronic equipment. The method isexemplary only; other steps may be added, steps may be removed, and thesteps may be performed in different orders than those shown, asappropriate. The temperature outside of the data center is monitored(step 602). When the temperature outside of the data center is not abovea predetermined value, the leaving air temperature can be maintainedbelow an inside setpoint by flowing cool water through a cooling systemwithin the data center (step 605). That is, water that is at least ascool as a predetermined water supply temperature is circulated throughthe data center. When the temperature outside of the data center risesabove the predetermined value, the leaving air temperature can beallowed to be greater than the inside setpoint by flowing water throughthe cooling system that is warmer than the predetermined water supplytemperature (step 606). The temperature can maintained below the insidesetpoint for more than 90% of the operating time of the electronicequipment.

Chilled water such as from a chiller may be provided through the modularcooling plant. The chilled water may be blended with cool water so thatthe inside setpoint may be either maintained or allowed to rise for onlya predetermined time. FIG. 7 is a flowchart 700 for cooling a datacenter having both chilled water and cooled water and using one or moreperiods of elevated temperatures. The method is exemplary only; othersteps may be added, steps may be removed, and the steps may be performedin different orders than those shown, as appropriate. The leaving airtemperature is monitored from an on-floor cooling unit in the datacenter (step 701). The external temperature is monitored (step 703). Itis determined whether the external temperature is above a predeterminedtemperature (step 705). If the external temperature is not above thepredetermined temperature, then cooled water is flowed through the datacenter (step 715). Alternatively, if the external temperature is above apredetermined temperature, then it is determined whether warm water hasbeen circulated through the data center for longer than a predeterminedtime (step 707). If warm water has not been circulated through the datacenter for longer than a predetermined time, then warm water iscontinued to be circulated through the data center (step 709).Alternatively, if warm water has been circulated through the data centerfor longer than a predetermined time, it is determined whether chilledwater should be circulated through the data center (step 711). Chilledwater is circulated through the data center (step 713).

Furthermore, although not shown, additional fans and local fan speedcontrol can be used to maintain the temperature inside the data center.Additional fans can be used to augment the fans in the trays and coolingunits. In one embodiment, the fans can collect air from a plurality ofracks. In another embodiment, fans can disperse cooled air to aplurality of racks. In another embodiment, a boost fan can be used tomake up for pressure drops in the plenums. For example, referring toFIG. 1, if a suction inlet to the fan 110 is much closer to the warm-airplenum 104 a than the warm-air plenum 104 b, an additional fan can beprovided in the path from the warm-air plenum 104 b so that theresulting static pressure in the warm-air plenum 104 a and the warm-airplenum 104 b is approximately equal. If the boost fan is not used,because of pressure drops traversing the plenum in the attic 105, thestatic pressures in the warm-air plenum 104 a and the warm-air plenum104 b could be sufficiently different to either require larger tray fansin the rack 102 b or could overdraw air through the rack 102 a. ydesigning a cooling system as described herein, the efficiency of thesystem can be improved. For example, allowing the temperature inside thedata center to rise above the inside setpoint for short periods of timecan minimize the need for chillers. Furthermore, using control valves toindividually control the flow of water to each on-floor cooling unitsuch that only the amount of water as is necessary is used, the totalamount of water required for the system is reduced by 10-50%. If theplant is designed to provide enough water for all control valves to bewide open all of the time, to provide water at a temperature below thepredetermined water supply temperature, and to provide for water lostdue to cleaning and/or evaporation, then the plant would need to havemore cool water available than necessary except for the warmest times ofthe year, for example for 30-40 hours per year. However, by allowing thewater temperature to rise during short periods and/or at higher flowrequirements, the plant can be designed to require less water. Likewise,cooling with relatively warm water may provide certain benefits whenchillers are used. In particular, when a chiller is allowed to provide asmaller temperature change to a coolant, e.g., water, the chiller mayprovide cooling for less electrical consumption per ton of coolant thanif it were required to impart a greater temperature change to thecoolant. By having elevated air temperatures in a cooled space, that is,the inside the data center and adjacent to the electrical equipment, thesupply water temperature may likewise be higher, and the need for achiller to cool the water may be less. As a result of these improvementsin efficiency and other design parameters discussed herein, the powerusage effectiveness (PUE) of the system, that is, the amount of powerentering the data center divided by the power used to run the electronicequipment or servers in the data center, can be less than 1.5, forexample less than 1.3, or about 1.2.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of what is described. For example, the steps of theexemplary flow chart on FIG. 6 may be performed in other orders, somesteps may be removed, and other steps may be added. Accordingly, otherembodiments are within the scope of the following claims.

1. A system for providing cooled air to a data center, comprising: adata center having electronic equipment, wherein the electronicequipment is in operation; a cooling water source; a plurality ofon-floor cooling units in the data center, each on-floor cooling unitconfigured to cool air warmed by a sub-set of the electronic equipmentin the data center; a plurality of proportioning control valves, eachproportioning control valve associated with a single on-floor coolingunit; and a controller, wherein each proportioning control valve isconfigured to turn in response to a signal from the controller, thevalve able to be closed, fully open or partially open, wherein when thevalve is closed, water is blocked from the on-floor cooling unit, whenthe valve is fully open, a maximum volume of water is circulated throughthe on-floor cooling unit, and when the valve is partially open, somepercent less than 100% of the maximum amount of water is circulatedthrough the on-floor cooling unit, and wherein the controller isconfigured to turn the corresponding control valve in response to achange in temperature.
 2. The system of claim 1, wherein the controlleris configured to turn the corresponding control valve in response to achange in a leaving air temperature of the corresponding on-floorcooling unit.
 3. The system of claim 2, wherein the controller isconfigured to turn the corresponding control valve such that the leavingtemperature of the corresponding on-floor cooling unit remains below aninside setpoint for at least 90% of the operating time of the system. 4.The system of claim 2, further comprising a plurality of sensors, eachsensor configured to measure the leaving air temperature of an on-floorcooling unit.
 5. The system of claim 1, wherein the proportioningcontrol valve acts independently of any other control valve.
 6. Thesystem of claim 1, wherein the cooling water source comprises a coolingtower.
 7. The system of claim 1, wherein each on-floor cooling unitcomprises a heat exchanger configured to transfer heat from theelectronic equipment to the cooling water source.
 8. The system of claim7, wherein the heat exchanger comprises coils located adjacent to one ormore common hot air plenums that receive heated air from the electronicequipment.
 9. A system for providing cooled air to electronic equipment,comprising: a data center having electronic equipment, wherein theelectronic equipment is in operation; a plurality of modules connectedby a first header, wherein each module comprises a plurality of on-floorcooling units in the data center and a cooling water source; and atleast one chiller connected by a second header such that the at leastone chiller is in fluid connection with more than one module in theplurality of modules.
 10. The system of claim 9, wherein the number ofchillers is less than the number of modules.
 11. The system of claim 9,wherein each on-floor cooling unit comprises a correspondingproportioning control valve configured to control the flow of waterthrough the on-floor cooling unit.
 12. The system of claim 9, whereinthe cooling water source comprises a cooling tower.
 13. The system ofclaim 9, wherein each on-floor cooling unit comprises a heat exchangerconfigured to transfer heat from the electronic equipment to the coolingwater source.
 14. The system of claim 13, wherein the heat exchangercomprises coils located adjacent to one or more common hot air plenumsthat receive heated air from the electronic equipment.
 15. A system forcooling air in a data center, comprising: a data center havingelectronic equipment, wherein the electronic equipment is in operation;a cooling water source configured to retain at maximum capacity a totalamount of water; and a plurality of on-floor cooling units in the datacenter, each on-floor cooling unit configured to cool air heated by aportion of the electronic equipment in the data center using water fromthe cooling water source; wherein the total amount of water isinsufficient to maintain a leaving air temperature of each on-floorcooling unit below an inside setpoint when a temperature outside of thedata center is above a predetermined external temperature.
 16. Thesystem of claim 15, wherein the cooling water source consists of acooling tower.
 17. The system of claim 15, wherein each on-floor coolingunit comprises a heat exchanger configured to transfer heat from theelectronic equipment to the cooling water source.
 18. The system ofclaim 17, wherein the heat exchanger comprises coils located adjacent toone or more common hot air plenums that receive heated air from theelectronic equipment.
 19. The system of claim 15, wherein: the system islocated in a geographical region; and for at least 90% of a year,temperatures outside of the data center are below the predeterminedexternal temperature.
 20. The system of claim 19, wherein for at least95% of the year, temperatures outside of the data center are below thepredetermined external temperature.